Analysis of Sentiment using Single Words

Sentiment Analysis using “bing” lexicon

The “bing” lexicon encodes words as either “positive” or “negative”. However, not all words used in the SONA speeches are in the lexicon so we need to adjust for this.

Sentiment per President

Let’s understand how many “positive” and “negative” words are used by each president across all their SONA speeches and create a metric called “sentiment” which is simply the total number of positive words minus the total number of negative words. We then adjust for the total number of words used from the lexicon in the “sentiment_score” metric.

## Joining, by = "word"

Sentiment Score per President

president	negative	positive	sentiment	sentiment_score
Zuma	788	1466	678	30.08
Ramaphosa	102	181	79	27.92
Mbeki	1314	2287	973	27.02
Motlanthe	180	263	83	18.74
Mandela	1036	1434	398	16.11
deKlerk	64	58	-6	-4.92

Insights:

Of the 3 presidents that have made more than 1 SONA speech, Zuma has the highest sentiment score, followed by Mbeki and then Mandela. Zuma’s sentiment score is nearly double Mandela’s. It’s interesting that the current President, Ramaphosa, has the second highest sentiment score, not far behind Zuma and only slightly ahead of Mbeki.

What are the 10 positive words most frequently used by each president?

## Joining, by = "word"

## Warning: `show.legend` must be a logical vector.

Which of the positive words most frequently used are common across presidents?

## Joining, by = "word"

## Joining, by = "president"

What are the 10 negative words most used by each president?

## Joining, by = "word"

## Warning: `show.legend` must be a logical vector.

How many of the negative words most used were used by each president?

## Joining, by = "word"

## Joining, by = "president"

What proportion of words used are positive vs negative?

## Joining, by = "word"

## Joining, by = "year"

Change in Sentiment over time

Is negative sentiment increasing over time?

## Warning in eval(family$initialize): non-integer #successes in a binomial
## glm!

## 
## Call:
## glm(formula = freq ~ as.numeric(year), family = "binomial", data = subset(sentiments_relative, 
##     sentiment == "negative"))
## 
## Deviance Residuals: 
##      Min        1Q    Median        3Q       Max  
## -0.21779  -0.05759   0.01224   0.07113   0.19692  
## 
## Coefficients:
##                   Estimate Std. Error z value Pr(>|z|)
## (Intercept)       25.88105  115.18655   0.225    0.822
## as.numeric(year)  -0.01316    0.05743  -0.229    0.819
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 0.39087  on 24  degrees of freedom
## Residual deviance: 0.33826  on 23  degrees of freedom
## AIC: 27.484
## 
## Number of Fisher Scoring iterations: 3

Is postive sentiment increasing over time?

## Warning in eval(family$initialize): non-integer #successes in a binomial
## glm!

## 
## Call:
## glm(formula = freq ~ as.numeric(year), family = "binomial", data = subset(sentiments_relative, 
##     sentiment == "positive"))
## 
## Deviance Residuals: 
##      Min        1Q    Median        3Q       Max  
## -0.19692  -0.07113  -0.01224   0.05759   0.21779  
## 
## Coefficients:
##                   Estimate Std. Error z value Pr(>|z|)
## (Intercept)      -25.88105  115.18655  -0.225    0.822
## as.numeric(year)   0.01316    0.05743   0.229    0.819
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 0.39087  on 24  degrees of freedom
## Residual deviance: 0.33826  on 23  degrees of freedom
## AIC: 27.484
## 
## Number of Fisher Scoring iterations: 3

Is average sentiment increasing over time?

## Joining, by = "word"

## 
## Call:
## glm(formula = avg_sentiment ~ as.numeric(year), family = "gaussian", 
##     data = sentiments_per_year)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -21.252   -7.877   -1.282    8.584   21.444  
## 
## Coefficients:
##                    Estimate Std. Error t value Pr(>|t|)  
## (Intercept)      -1369.9476   652.4104  -2.100   0.0460 *
## as.numeric(year)     0.6952     0.3253   2.137   0.0425 *
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for gaussian family taken to be 153.4216)
## 
##     Null deviance: 4536.4  on 26  degrees of freedom
## Residual deviance: 3835.5  on 25  degrees of freedom
## AIC: 216.44
## 
## Number of Fisher Scoring iterations: 2

## 
## negative positive 
##     4782     2006

Distribution of “bing” Sentiment per President

## Joining, by = "word"

Change in “bing” Sentiment over time

## Joining, by = "word"

## Warning in simpleLoess(y, x, w, span, degree = degree, parametric =
## parametric, : Chernobyl! trL>n 6

## Warning in simpleLoess(y, x, w, span, degree = degree, parametric =
## parametric, : Chernobyl! trL>n 6

## Warning in sqrt(sum.squares/one.delta): NaNs produced

## Warning in stats::qt(level/2 + 0.5, pred$df): NaNs produced

Sentiment Analysis using “afinn” lexicon (scale from -5 negative to +5 positive)

## Joining, by = "word"

afinn Sentiment

score	n	weighted_score
-5	2	-10
-4	27	-108
-3	689	-2067
-2	1059	-2118
-1	867	-867
1	2422	2422
2	3525	7050
3	379	1137
4	42	168
5	32	160

## 
##  -5  -4  -3  -2  -1   0   1   2   3   4   5 
##  16  43 264 965 309   1 208 448 172  45   5

Distribution of “afinn” Sentiment per President

## Joining, by = "word"

Change in “afinn” Sentiment over time

## Warning in simpleLoess(y, x, w, span, degree = degree, parametric =
## parametric, : Chernobyl! trL>n 6

## Warning in simpleLoess(y, x, w, span, degree = degree, parametric =
## parametric, : Chernobyl! trL>n 6

## Warning in sqrt(sum.squares/one.delta): NaNs produced

## Warning in stats::qt(level/2 + 0.5, pred$df): NaNs produced

Sentiment Analysis using “nrc” lexicon (infers emotion with certain words)

## Joining, by = "word"

## Joining, by = "president"

Let’s check the distribution of all “nrc” words:

## 
##        anger anticipation      disgust         fear          joy 
##         1247          839         1058         1476          689 
##     negative     positive      sadness     surprise        trust 
##         3324         2312         1191          534         1231

Sentiment Analysis using “nrc” lexicon

## Joining, by = "word"

Step 4: Extracting insights

####Understanding the topics via the bigrams

The beta matrix produced gives the probability of the topic producing that bigram (i.e. that the phrase is in reference to that topic.) From this measure, one can get a sense of what the character of the topic is. By using the most popular phrases in each topic, understanding of the flavour of each topic emerges. However, it must be kept in mind that terms can belong to more than one topic so applying some logic to get a theme or flavour should be done liberally.

#####Topic One

From the display of popular terms, it can be determined that the topic one has a vague connection to “job creation” - this is the most common terms but is supported by other terms that have a high probability of being generated by this topic such as: + “world cup” + “national youth” + “infrastructure development”

These concepts all support the idea of the job creation as each of these will generate jobs for the country. But there is some noise in the topic for “address terms” i.e. honourable speaker or honourable chairperson. “Nelson Mandela” and “President Mandela” crop up too which suggests that alongside the job creation theme, there exists some of what can be termed “terms of endearment”

#####Topic Two

As with the previous topic, there is some random “terms of endearment” in this topic as well (i.e. “madam speaker”) but it is not as evident as in the first topic. This is to be expected as bigrams can be generated by more than one topic as each topic is a mixture of bigrams! The next four terms sums out the main themes for this topic: + “Economic Empowerment” + “Black Economic” + “Justice System” + “Criminal Justice”

In summary, this topic can be summed as “Economy/ Criminal and Justice System”

#####Topic Three

Despite the most popular terms being “United Nation” and “private sector”, there a theme that is “developing”. As in development plan, resource development, national development and development programme etc. And thus, the topic is named.

#####Topic 4

Once again, there is a “term of endearment” in the popular terms (“fellow south” which is assumed short for “fellow South African’s” which is one of former President Zuma’s favourite phrases). With all the other terms combined, a theme of “Social Reform/ Regional and Municipal Government” takes shape.

Given that there is a possible trigram evident here, it may be worth exploring in future work.

#####Topic 5

“Public sector” and “private sector” are popular terms in topic 5. After consideration of the various other terms, of which some have cross over with other topics and discussion, the eventual name for this topic became “Public Sector Entities”

A different way of looking at this topic could be to investigate the biggest differential in terms between topics. For instance, using the log(base 2) ratio between topic 1 and topic 5, shows the terms that have the widest margin between the two topic (i.e. are far more likely to be in topic 5 versus topic 1)

For instance, “social programmes”, “human fulfilment”, “rights commission” are all generated in significantly larger proportions by Topic 5 compared to Topic 1 while “national social”, “training colleagues” and “sector unions” all exist with in Topic 1.

Given the naming of Topic 5 as “Public Sector Entities” and Topic 1 as “Job Creation/Terms of Endearment” these terms do seem to be grouped in line with expectation.

####Understanding the mixture of topics within the sentence

The LDA model allows each of the sentence to be represented as a mixture of topics. The gamma matrix shows the document-topic probability for each sentence. i.e. the probability that each sentence is drawn from that topic. For instance, the follow sentence sampled from random shows that it has a 0.905% probability of being drawn from topic 4 based on the use of the bigrams within it. The sentence appears to be talking about the water and the infrastructure around it. The label for topic of was “Social Reform/ Regional and Municipal Government” and this statement seems to be somewhat relevant to it.

Sample sentence showing topic probabilities

president	year	sentence	X1	X2	X3	X4	X5
Zuma	2010	yet, we still lose a lot of water through leaking pipes and inadequate infrastructure.	0.023575	0.023575	0.023575	0.9057001	0.023575

Using this method, the sentences can be roughly classified to a topic based on the probabilities (i.e. classify the sentence by the topic with the highest probability) and further analysis can be conducted.

(Note: the which.is.max breaks ties at random so where a sentence has equal probabilities, is will decide at random to which topic it gets assigned)

Consider the mixture of topics that each individual president covers during the SONA address. Despite the imbalance in the number of sentences said by each president, there seems to be a fairly standard shape to the topics discussed. The two exceptions to this are de Klerk and Zuma. All other presidents tend to send around 10-15% on topic 1 (“Job Creation/Terms of Endearment”) and the 15-20% on Topic 2 (“Economy/Criminal and Justice System”), Topic 3 (“Development”) and Topic 4 (“Social Reform/Regional and Municipal Government”) and the around another 10% on Topic 5 (“Public Sector Entities”). This trend means that it may be difficult for a supervised model to pick up difference in presidents based on the topic covered.

As stated, the only two presidents where this trend differs are President de Klerk and President Zuma. President de Klerk spend the majority of his time on Topic 1 (“Job Creation/Terms of Endearment”) followed by Topic 2 (“Economy/Criminal and Justice System”). Given the context around the time period, it may be unsurprising that “terms of endearment” and “criminal and Justice systems” come up since his speech would be littered with names of people and political parties as well as talking about past injustices.

President Zuma spends the majority of his speeches on Topic 4 (“Social Reform/Regional and Municipal Government”). Once again, given context that his terms as President was marked with service delivery strikes, two major droughts over a few different regions and discussions around and reform this may be unsurprisingly. And in fact, when the most popular terms from topic 4 is recalled (“fellow south”) is may even be predictable that this would be the most “talked about” topic for President Zuma. What is interesting that given the attention to the issues of State Capture that characterised Zuma’s presidency, his coverage of Topic 5 (“Public Sector Enterprises”) is much smaller than that of his peers.

A similar analysis can be taken over time.

The graph shows that over time, topics 1 and 5 are the least discussed topics while topics 2,3 and 4 all get much the same airtime. There are a number of notable spikes/valleys: + In 1996, Topic 2 (“Economy/Crime and Justice System”) spikes

The 1996 SONA was a few months ahead of the introduction of the new constitution as well as at the time of the start of the Truth and Reconciliation Commission. It could be suggested that these two ideas would drive up the topic in the SONA speech.

• In 2005, topic 1 (“Job Creations/Terms of Endearment”) dives while topic 4 (“Social Reform/Region and Municipal Government”) and Topic 2 (“Economy/Criminal and Justice System”) spike considerably

  Mbeki’s term in presidency (1998 - 2008) was characterised by a rise in crime specifically in farm attacks as well as the HIV/AID epidemic and the start of the Black Economic Empowerment in 2005 which could attribute the spikes and drops of topics in 2005.

• In 2012, Topic 2 dives considerably (“Economy/Criminal and Justice System”)

From various media reports, Zuma’s 2012 SONA speech largely covered the success of the government while skipping over future plans. It may be a reason while Topic 4 (“Social Reform/Regional and Municipal Government”) rises sharply.

####Step Five: Using the topic to predict the president

One of the aims behind topic modelling is to reduce the dimensions of the data to allow for other techniques to be applied. In this instance, the aim was to reduce the SONA speeches to a collection of topics that would help predict which president was responsible for a sentence in the SONA speech. The assumption was that each president might have a unique set of topics or mixture of topics that could characterise their particular speech. However, there does not seem to be evidence of this. The matrix with the probability of each sentence belonging to a topic is used in Section X and the results are discussed.

Neural Net with Bag of Words Data

The count of each word that have been used in each sentence is what we are going to be feeding in. We need to unnest the sentence data, count each word in each sentence and spread the sentence word counts so that we have sentence id’s in each row and we have each word as column id’s. This is the simplest neural net model that we can try, so that was our first model.

This model has L2 regularization to avoid overfitting but even so it didn’t help very much. The accuracy is 0.55886. The optimizer_rmsprop has learning rate 0.003. This was choosen after trying lr=c(0.001, 0.002, 0.003) To make readability easier the model with best learning rate is used.

As we can see from the plot model overfits after the second iteration, since the loss function start increasing in value, so to avoid that let’s use a smaller model with less neurons and add a dropout.

180, 50, 57, 14, 7, 1, 33, 181, 16, 8, 12, 2, 56, 26, 111, 10, 4, 3, 7, 4, 3, 3, 0, 3, 4, 9, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1

Accuracy rate is: 0.5911.

Neural Net with tf-idf Data

TFIDF is a statisic that shows how important a word is to it’s corpus. So if we are feeding NN with TFIDF we are logically expecting the results to be slightly better than the word-cout NN model.

1	2	3	4	6
183	46	51	0	0
48	203	18	1	0
52	17	118	1	0
16	8	11	0	0
5	16	3	0	0
0	3	4	2	1

Accuracy rate is: 0.6245.

Neural Net with Sentiment Analysis Data

1	2	3
143	134	3
70	199	1
91	95	2
15	20	0
8	15	1
3	7	0

Accuracy of sentiment analysis is: 0.426 [0.4263].

Neural Nets with Topic Modelling Data (Gamma Values)

Topic modelling only predicted for president 1(Mbeki) and president 2(Zuma).

CNN with Transfer Learning (Pre-trained Embeddings)

We will be using GloVe embeddings. GloVe stands for “Global Vectors for Word Representation” and is an unsupervised learning algorithm for obtaining vector representations for words as it is stated in their website. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space. [Reference:Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation:https://nlp.stanford.edu/pubs/glove.pdf ] Specifically, we will use the 100-dimensional GloVe embeddings of 400k words computed on a 2014 dump of English Wikipedia.

This note is taken from the reference given above and as they state the accuracy they achieved on python is twice as good.

“IMPORTANT NOTE:This example does yet work correctly. The code executes fine and appears to mimic the Python code upon which it is based however it achieves only half the training accuracy that the Python code does so there is clearly a subtle difference. We need to investigate this further before formally adding to the list of examples”

[reference for implementation on Python: https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html] [reference for implementation on R: https://keras.rstudio.com/articles/examples/pretrained_word_embeddings.html] [reference for implementation on R: https://github.com/rstudio/keras/blob/master/vignettes/examples/pretrained_word_embeddings.R]

Also for the pre-trained embeddings to work well it needs to be trained with similar type of data that you are trying to classify, and the fact that the GloVe embeddings are trained on Wikipedia data one can expect that it would not neccesarily help predict the presidents better for our sentences.

Majority of the sentences of Mbeki is predicted as Mandela(379/569). Majority of the sentences of Zuma is predicted as Mandela (244/534) and the secon majority is predicted as himself (195/534). Majority of the sentences of Mandela is predicted as Mandela(263/381). Majority of the sentences of Mothlane(38/66) is predicted as Mandela. Majority of sentences of Ramaphosa is predicted as Mandela(19/47) or Zuma(15/47). Mojority of the sentences of deKlerk is predicted as Mandela(14/17).

Evaluate Out of Sample Performance

test_acc

Evaluate Out of Sample Performance